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Abstract for Evaluation of Amazonia Lee Reading Intervention in Peru 
Background 


Countries in Latin America and the Caribbean have made progress in increasing primary school 
enrollment and completion among the very poor. However, challenges in improving reading 
persist. This study will contribute to the evidence base on what works to improve early-grade 
reading skills by evaluating an early reading program implemented in two departments of the 
Amazon region of Peru: Ucayali and San Martin. 


Intervention 


Amazonia Lee has four main components. First, the capacity development component aimed to 
strengthen pedagogical skills and instructional capabilities of teachers, principals, and regional 
specialists. The main mechanism is conducting teacher training workshops based on the balanced 
approach. The workshops are targeted to first, second, and third grade teachers and aim to 
improve reading instruction. The balanced approach combines evidenced-based best practice 
pedagogical skills with the communicative textual approach to reading supported by the Peruvian 
Ministry of Education. This component also uses in-class teacher coaching and demonstration 
classes, teacher study groups, leadership training, and improvement and delivery of materials. 


The following three components were originally envisioned to be implemented at the regional 
level in both departments. However, during the period of the study they were only implemented 
in the schools implementing Amazonia Lee. First, the community engagement component aimed 
to promote the active engagement of the educational community in support of student learning. 
Second, a regional assessment system aimed to monitor student achievement and to develop 
local capacity related to these assessments. Last, a teacher non-monetary incentive system 
aimed to motivate teachers to improve student learning. 


Research Question and Design 
The evaluation addressed the following central question: 


e What was the impact of Amazonia Lee programs on early-grade reading and other 
intermediate outcomes relative to other prevailing practices and programs? 


We randomly assigned eligible schools in Ucayali and San Martin to one of two groups: the 
treatment group was offered all program components of Amazonia Lee, and the control group 
was not offered this program. The evaluation sample includes 70 schools in Ucayali and 200 in 
San Martin. In each region, half of the schools were randomly assigned to the treatment group 
and half to the control group. 


In Ucayali, control schools received the usual services the Ministry of Education provides. In San 
Martin, a majority of the control schools received additional assistance in the form of a program 
known as Soporte Pedagogico (Pedagogical Support), a Ministry of Education initiative that aims 
to improve student learning in all content areas. Soporte Pedagdgico aims to strengthen 
teachers’ skills, provide after-school tutoring for at-risk students, deliver educational materials, 
and improving the school management capacity. Thus, in San Martin, we are comparing 
outcomes from schools receiving the Amazonia Lee intervention to schools receiving Soporte 
Pedagogico. 


This evaluation measures the effects of exposure to the program during first and second grade. 
In each region, we estimated the impacts of Amazonia Lee activities on student reading outcomes 
at the end of 2nd grade by comparing the average outcomes of students in the treatment schools 
to those in the control schools. The impact estimates account for students’ baseline performance 
on an emergent literacy assessment administered when students were starting 1st grade. We 
also controlled for student background characteristics and school characteristics, such as the 
schools’ physical infrastructure. 


Impacts on teacher outcomes were estimated by comparing 2nd-grade teachers in the treatment 
and control groups. 


Data Collection 


The main data sources for the study are student reading assessments, classroom observations, 
teacher surveys, and school infrastructure surveys. All of these instruments were collected at 
endline in 2016 and at baseline in 2015 or 2014. 


e Individual reading assessment. We designed and piloted an Early Grade Reading 
Assessment consisting of five tasks: oral comprehension, familiar word reading, decoding, 
reading fluency, and reading comprehension. The individual assessment was collected on 
a longitudinal sample of students at two points in time. At baseline, in each study school, 
we randomly selected one 1st-grade classroom and within it a random sample of 1st- 
grade students (approximately 10). At endline, we tracked the students in our study 
sample to evaluate their reading skills. 


e Classroom observation. We developed a classroom observation instrument to be fielded 
in the reading and communication class. The instrument focused on measuring teachers’ 
use of class time on different types of literacy-related activities, teachers’ literacy 
instruction, teachers’ use of general pedagogical practices, and quality of the classroom 
literacy environment. The classroom observation was collected at baseline on a random 
sample of 1st-grade classrooms (one per school) and similarly on a random sample of 2nd- 
grade classrooms at endline. 


e Teacher survey. We also administered a teacher survey to all teachers whose classrooms 
were observed. Teacher surveys gathered information on use of instructional practices, 
participation in professional development, education and experience, occupational 
needs, career expectations, and background characteristics. 


Findings 


Amazonia Lee services reached the treatment groups in both departments. The primary 
interventions of the program were concentrated on training, in-class coaching, and provision of 
materials. 


Teachers in both departments seem to be implementing Amazonia Lee. |n Ucayali, Amazonia 
Lee had positive impacts on explaining the use of different kinds of text, reading daily for 
pleasure, and teaching the use graphic organizers—practices consistent with the balanced 
approach. In San Martin, Amazonia Lee had positive impacts on practicing graphomotor skills, 
teaching punctuation, explaining the use of different texts, and developing vocabulary—practices 
consistent with the balanced approach. 


In Ucayali, Amazonia Lee had positive and substantial impacts on reading outcomes. We found 
an impact of 0.25 standard deviations on reading familiar words and on decoding, 0.16 standard 
deviations on fluency, and 0.28 standard deviations on reading comprehension. 


In San Martin, students in the treatment group had similar reading outcomes to students in the 
control group. We found no differences between groups on oral comprehension, words read 
correctly, fluency, decoding, and reading comprehension scores. 


Conclusions 


When compared to prevailing practice, Amazonia Lee achieved substantive effects of magnitudes 
similar to those found in other rigorous evaluations. However, when compared to a program that 
provided services similar to those provided by Amazonia Lee, students in both groups had similar 
reading outcomes. 


Additional Tables (not included in the word count) 


Table 1. Amazonia Lee l|led to positive impacts of substantive magnitude on 
reading familiar words, decoding, and reading comprehension in Ucayali 


Treatment Control Effect 

adjusted mean adjustedmean Difference p-value sizes 
Number of correct oral comprehension 2.7 2.6 0.1 0.299 0.09 
responses 
Number of familiar words read correctly ta 6.0 el” 0.011 0.25 
Number of pseudo-words read correctly 16.8 13.3 3.5* 0.001 0.25 
in one minute, decoding 
Number of words read correctly in one 34.7 30.1 4.5* 0.035 0.16 
minute, fluency 
Number of correct answers in reading 2.5 1.9 0.6* 0.001 0.28 


comprehension 
Number of students 380 360 


Source: Amazonia Lee Child assessment—Final Follow-Up 2016. 


Note: Regression-adjusted means ordinary least squares and include covariates to account for the design (strata 
variables). We also include baseline score of students and some student’s characteristics (2015) and 
baseline infrastructure characteristics of schools (2014). Effect sizes were calculated before rounding the 
adjusted difference. 


* Difference in group means is statistically significant at the 0.05 level. 


Table 2. Students reading outcomes were similar between groups in San 
Martin 


Treatment Control Differenc Effect 
adjusted mean adjusted mean e p-value sizes 
Number of correct oral comprehension 2.9 2.9 0.1 0.315 0.04 
responses 
Number of familiar words read 8.1 8.2 -0.1 0.441 -0.03 
correctly 
Number of pseudo-words read 19.6 20.2 -0.6 0.175 -0.05 
correctly in one minute, decoding 
Number of words read correctly in one 42.9 44.1 -1.3 0.129 -0.05 
minute, fluency 
Number of correct answers to reading 3.0 3.0 0.0 0.775 0.01 
comprehension questions 
Number of students 824 822 
Source: Amazonia Lee Child assessment—Final Follow-Up 2016. 
Note: Regression-adjusted means ordinary least squares and include covariates to account for the design (strata 


variables).We also include baseline score of students and some student's characteristics (2015) and 
baseline infrastructure characteristics of schools (2014). Effect sizes were calculated before rounding the 
adjusted difference. 


